A mixed reality (MR) system provides to the user a mixed reality space image obtained by combining a physical space image and a virtual space image. They are generated in correspondence with the viewpoint position, line of sight direction, and the like, of the user. The MR system can present to the observer as if virtual objects ware existing in a physical space, and allows the observer to make an observation with a sense of actual dimensions and higher reality than a conventional virtual reality (VR) system.
On the other hand, designs (shapes and designs) using three-dimensional (3D) CAD have become mainstream in the design and manufacture fields. In this case, as a method of evaluating an object designed by 3DCAD, a method which displays data (solid format) created by the 3DCAD on the screen of a computer as a 3DCG and visually evaluates it, is commonly used.
In this case, a method of allowing the user to observe 3DCG data using an object manipulating device by combining both the aforementioned techniques, i.e., the above MR system and a 3DCAD technique is available. In order to set a manipulation function, the following method is used. A control panel is displayed within the MR space observed by the user and icons displayed on that panel are selected by a manipulation unit such as a stylus or the like.
An observation of the MR space is realized in four steps. First, by measuring a viewpoint position/direction and the like of a user in real time. Second, by acquiring a video of a physical space captured at the viewpoint position/direction by using a video camera, or the like. Third, by composing the video and a virtual object (CG object) viewed from the viewpoint position/direction of the user. Fourth, by displaying the composed result.
In order to measure a position and direction of a viewpoint of the user in real time, a system (a 6 degrees of freedom sensor, herein), which measures or estimates 6 degrees of freedom of a position and orientation, is used. For example, in the case of using the general measurement system of a position and orientation, 6 degrees of freedom of a position and orientation of a receiver having a size of a few centimeters square can be measured.
By associating the measured 6 degrees of freedom with a position and direction of a virtual camera in a virtual space, the MR space can be observed.
Upon operating a CG object in the MR space, the method of associating a position and orientation of a 6 degrees of freedom sensor different from a sensor used to detect a viewpoint and line of sight of a user with an object coordinate system of the CG object, by using the 6 degrees of freedom sensor, is disclosed. A CG object moves and rotates upon moving and rotating of a receiver by associating the measured value with the object coordinate system.
By the method above, the MR space that is generated from a physical space including virtual objects can be observed. However, it is problematic because a virtual object always superimposes on a physical object even if the physical object is in front of the virtual object.
In order to solve the above problem, there is a method for preparing a CG model of a physical object, which utilizes transparency for a physical image and opacity for a CG image. By using this method, the problem can be solved. However, a CG model of a physical object must be prepared. The method is suitable for a stationary object such as a desk and a chair arranged in a physical space in advance. However, the method is unsuitable for an object such as the hand of a user. Because it must measure the position and shape of “hand of a user”, changing its shape freely in real time and then generate a model thereof. It is very difficult or impossible to measure the shape of an object changing dynamically in real time and generate a model thereof.
In order to cope with superimposition of the hand of a user, there is a technique of hand over lay. According to this technique, CG is not drawn on an area of a user's hand within an image captured by a camera, or a physical image of user's hand is drawn on an image generated based on conventional art. There are some techniques for extracting an area of a hand, for example only hand color (flesh color) can be extracted by performing chromakey processing.
However, in the MR system, when the user is about to select a CG icon in a virtual environment using an actual pointing device, he or she cannot easily recognize a relative positional relationship such as depth ordering or the like, and it is difficult for him or her to select such icon.
Furthermore, upon making such operation in the MR space, the problem of superimposition of a hand occurs. However, even using the conventional hand over lay method in order to solve that problem, a shadow of a physical object such as a hand cannot be clearly expressed.